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ABSTRACT 

New  sensors,  communication  infrastructure,  and  information  systems  will  make  plenty  of  information 
about  actual  mission  scenarios  available.  Innovative  concepts  like  Network  Centric  Warfare  (NCW) 
describe  the  consequent  availability  of  this  information  for  each  command  level  at  any  time,  at  any  place. 
But  the  huge  amount  of  information  may  also  lead  to  information  overload  for  the  human  commander: 
Relevant  information  may  be  overseen.  This  might  results  into  building  a  wrong  mental  model  of  the 
mission  scenario  and  wrong  decisions.  Therefore,  the  development  of  new  display  paradigms  and 
technology  for  information  presentation  is  essential.  They  should  be  based  on  established  procedures  in 
military  decision  making  and  staff  planning  and  allow  the  easy  transfer  of  them.  This  article  presents 
some  example  approaches  of  applying  Augmented  Reality  (AR)  and  Virtual  Environment  (VE)  technology 
as  advanced  Tactical  Situation  Display  (TSD)  for  different  military  command  levels.  It  is  concluded  that 
research  on  microergonomic  (i.e.  perceptual,  motor  capabilities  for  data  visualization  and  data  input)  and 
macro  ergonomic  topics  (i.e.  workload,  situational  awareness,  decision  making  performance)  is  needed  for 
a  reasonable  application  ofVE  as  a  display  of  C4ISR  systems. 


1  TACTICAL  SITUATION  DISPLAYS  (TSD) 

The  main  function  of  a  TSD  is  to  display  the  current  situation  and  location  of  own  and  reconnoitred 
oponent  troops  and  facilities.  It  is  used  for  tactical  planning  of  intended  future  operations.  Quantity  and 
quality  of  situation  data  are  essential  for  an  adequate  operation  planning.  A  basic  TSD  consists  of  scaled 
maps  of  the  regional  properties  of  the  mission  area  displaying  all  relevant  information  and  intentions.  It 
allows  a  brief  overview  about  tactical  and  regional  situation  for  the  specific  mission  commander  and 
higher  command  levels.  The  main  parts  of  the  TSD  are:  Regional  situation,  including  geo-referenced 
terrain  and  feature  data;  forces’  strengths  and  locations;  missions’  sections  and  foci;  areas  for  mission 
support  and  control.  This  information  is  usually  coded  by  a  special  tactical  symbology. 

1.1  Today’s  Tactical  Situation  Displays 

Today  there  are  mainly  two  common  types  of  TSDs.  The  first  one,  shown  in  figure  1,  is  a  command  post 
in  the  field.  The  TSD  used  here  works  by  means  of  map,  paper  and  pencil.  Actual  information  is 
transmitted  by  radio  or  field  telephone  and  manually  drawn  into  a  map.  It  is  obvious  that  in  time-critical 
processes  with  large  amounts  of  rapidly  changing  information  an  overload  of  the  operators  is  very  likely. 
The  display  may  not  show  actual  or  valid  information  then.  It  is  likely  that  the  operators  will  be  busy  with 
keeping  the  display  up-to-date  instead  of  decision  making.  However,  such  a  display  has  the  advantage  of 
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being  fielded  so  that  the  commander  is  in  the  field:  This  will  induce  situational  awareness.  The 
commander  experiences  the  terrain  elevation,  terrain  feature,  weather,  etc.  and  knows  “what  is  really 
going  on”  during  the  mission. 


Figure  1:  TSD  at  Command  Posts  “n  the  field”. 


On  the  other  hand  there  are  TSDs  at  operation  centers,  as  can  be  seen  in  figure  2.  The  centers  are  usually 
more  ore  less  distant  from  the  mission  so  that  commanders  are  not  put  at  risk  of  combat.  Tactical  situation 
data  is  pre-processed  and  computers  are  used  to  visualize  the  results.  The  advantages  of  these  TSD’s  are 
(1)  actuality  of  data,  provided  that  the  communication  infrastructure  is  fast  enough,  (2)  different  views  of 
levels  of  data  aggregation,  and  (3)  possibilities  to  include  additional  mission  information. 


Figure  2:  TSD  at  (provisionary)  Operation  Centers. 

However,  one  disandvantage  of  these  TSDs  is  a  very  abstract  presentation  of  mission  data.  Today’s 
displays  apply  two-dimensional  monitors  or  projection  walls,  which  are  sometimes  difficult  to  operate. 
Display  of  information  (projection)  and  planning  environment  (card  table)  is  often  not  the  same  and 
requires  additional  information  processing.  Consequently,  a  combined  mission  display  and  planning 
environment  should  be  achieved.  Displays  shoul  make  better  use  of  the  human  perceptive,  cognitive,  and 
motor  ressources. 

1.2  Future  Mission  Scenarios  and  Requirements 

Future  military  scenarios  will  be  characterized  by  a  diversity  of  mission  objectives  and  tasks.  They  have 
changed  from  a  purely  military  domain  to  areas  like  peace  supporting  and  peace  keeping  operations  as 
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well  as  humanitarian  aid  in  a  widely  unknown  area  and  environment.  Actual  concepts  like  Network- 
Centric  Warfare  (NCW)  include  huge  amounts  of  highly  dynamic  information,  which  will  be  available  in 
any  imaginable  scenario  [1].  Although  data  processing  might  become  automated  to  a  special  extend, 
decisions  will  still  be  made  by  human  operators.  This  requires  advanced  display  and  support  systems  for 
the  decision  making  process.  They  have  to  support  the  human  commander  in  cumulating  a  variety  of 
diverse  information  for  creating  a  correct  mental  model  of  the  current  situation.  With  this  massive  amount 
of  rapidly  changing  information,  relevant  information  may  be  overseen  in  time-critical  situations.  These 
perceptual  errors  can  easily  result  into  a  wrong  situational  mental  model  and  consecutive  wrong  decisions. 
It  is  not  necessary  to  point  out  that  wrong  decisions  may  easily  lead  to  extensive,  sometimes  fatal 
consequences.  It  is  necessary  to  apply  advanced  system  interfaces  that  make  use  of  the  human  perceptual, 
cognitive,  and  motor  capabilities.  This  way  the  amount  of  transferred  information  can  be  enhanced  and  the 
understanding  of  complex  mission  scenarios  is  supported. 

When  examining  the  applicability  of  special  display  technology  there  are  several  requirements  to  be 
considered.  They  refer  to  a  functional  and  technical  context. 

A  functional  requirement  is  the  consideration  of  the  mission  context  and  responsibility  of  the  user.  Each 
user  or  commander  has  special  responsibilities  and  a  designated  role  for  mission  execution.  He  should  be 
provided  with  the  important  information  only  and  not  be  distracted  by  other.  This  concept  is  further 
specified  by  the  Common  Relevant  Operational  Picture  (CROP).  CROP  is  a  presentation  of  timely,  fused, 
accurate,  and  relevant  information  that  can  be  tailored  to  meet  the  requirements  of  the  commander  [2]. 
Even  though  actual  missions  are  often  carried  out  with  a  “slim”  command  hierarchy,  each  commander  still 
has  designated  responsibilities.  The  concept  of  need-to-know  has  to  be  considered  when  selecting  relevant 
information  and  type  of  presentation.  Higher  command  levels  are  usually  not  interested  in  detail 
information,  e.g.  special  terrain  and  vegetation  features.  But  these  would  be  important  for  low  command 
levels  and  platoon  commanders. 

Another  functional  requirement  is  the  intention  and  goal  when  using  the  display  in  decision  making  and 
mission  planning.  An  obvious  intention  would  be  to  collect  relevant  information  about  the  actual  mission. 
This  requires  a  simple,  one-way  information  transfer  only.  But  often  decision  making  requires  additional 
mission  analysis  and  supporting  functionality.  In  this  case  interaction  metaphors  and  techniques  have  to  be 
applied. 

In  addition  to  these  functional  requirements  there  are  several  technical  requirements.  Usually  high-level 
decision  making  does  take  place  in  stationary  command  posts  with  a  lot  of  IT-infrastructure  and 
communication  lines.  In  contrast  to  this,  low-level  decision  making  is  done  in  a  distributed,  operational 
scenario  with  few  portable  or  no  IT-infrastructure.  These  settings  result  into  different  requirements  for 
portability  and  mobility  of  the  display.  Another,  more  practical  requirement  is  the  light  situation  of  the 
display’s  environment.  Only  few  projection  technologies  would  work  under  daylight  conditions;  most 
require  a  darker  environment.  There  are  several  other  technical  requirements  on  safety,  robustness,  etc. 
exists  which  limit  practical  applicability  of  AR  and  VE  technology  as  TSD.  They  have  to  be  considered  in 
addition  to  special  visualization  and  interaction  characteristics. 


2  APPLICABILITY  OF  VE-TECHNOLOGY  FOR  C4ISR 

There  are  various  studies  about  the  applicability  of  AR  and  VE  -technology  in  C4ISR.  They  can  be 
divided  into  two  large  groups.  The  first  group  consists  of  concepts  and  long-term  programs  including  AR 
and  VE.  It  is  a  high  level,  political  approach,  which  is  very  mission-oriented.  The  second  group  is 
characterized  by  more  concrete  AR  and  VE-projects  of  laboratories.  The  approaches  of  this  group  are 
typically  technology-oriented. 
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The  Swedish  system  ROLF  would  be  part  of  the  first  group.  Its  goal  was  to  determine  new  possibilities  of 
applying  VE-technology  for  mobile  command  posts.  It  included  a  semi-immersive  VE-system  as  a  tactical 
situation  display.  The  display  was  used  to  visualize  positions  of  own  and  opponent  troops,  locations  of 
important  institutions,  and  terrain  and  weather  data  from  different  viewpoints.  Additional  data  processing 
was  used  to  assure  that  only  important  information  was  visualized  to  prevent  distraction  of  the  operator 
[3].  Another,  long-term  program  is  the  Command  Post  of  the  Future  Program  (CPoF)  of  D  ARP  A  [4].  The 
overall  goal  of  this  program  was  to  foster  the  decision  making  process  with  ongoing  reduction  of  the  staff. 
The  program  included  an  interactive,  three-dimensional  visualization,  three-dimensional  interaction  with 
computer-generated  objects,  presentation  of  inaccuracy  and  probability,  integration  of  dynamic  factors, 
three-dimensional  symbolic,  integration  of  natural  language  processing  and  integration  of  knowledge- 
based  systems. 

The  second,  more  technology-oriented  group  of  approaches  is  more  common.  Institutions  and  laboratories 
working  in  this  area  use  a  variety  of  different  AR  and  VE-technologies.  The  setups  are  often  reconfigured 
for  different  research  projects  and  analysis.  For  instance,  the  US  Naval  Research  Laboratory  (NRL)  has 
developed  an  advanced  battle  planning  and  management  system.  The  system  included  a  semi-immersive 
display  and  multi-modal  interaction.  It  was  found  to  be  suitable  for  virtual-prototyping,  interactive  mission 
planning  and  increasing  situational  awareness  [5].  Similar  approaches,  like  Mirage  of  the  Army  Research 
Lab  (ARL)  [6],  the  Visualization  Architecture  Technology  (VAT)  of  the  Crewstation  Technology 
Laboratory  (CTS)  [7]  or  the  Electronic  Sand  Table  of  MITRE  Corp.  [8]  also  applied  projection-based  VE- 
technology  as  described  in  a  later  chapter  of  this  paper.  Other  approaches  applied  individual  VE  [9,  10). 
But  despite  these  operational  approaches,  education  and  training  in  this  field  is  still  using  traditional 
media. 

2.1  Command  levels  and  proposed  VE-displays 

Based  on  the  functional  and  technical  requirements  of  section  1.2  we  have  proposed  the  application  of 
various  TSD  systems  and  technologies  for  the  different  command  levels  as  shown  in  figure  3. 
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Figure  3:  Command  Level  and  Proposed  TSD  System. 


The  displays  specified  here  are  characterized  by  special  functionality,  portability  and  interactivity.  On 
setting  up  this  sketch,  it  was  postulated  that  decisions  on  lower  command  levels  require  mobile  or  portable 
systems,  i.e.  AR  systems.  More  massive,  stationary  displays  can  be  used  with  higher  command  levels. 
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Moreover,  a  darker  light  environment  is  usually  difficult  to  achieve  for  lower  command  level,  whereas  it  is 
no  problem  in  operation  centers. 

2.2  VE-Tactical  Situation  Presentation 

For  higher  command  levels  the  TSD  is  used  to  display  relevant  data  and  support  the  presentation  of  the 
tactical  situation  for  expert  staff  members.  Therefore,  a  presentation  metaphor  can  be  applied.  A  vertical 
3D-stereoscopic  projection  can  be  used  as  a  display.  It  visualizes  geographic  terrain  data  and  tactical 
situation  data.  The  terrain  data  is  derived  from  military  terrain  and  elevation  data,  which  is  available  in 
high  resolution  for  relevant  areas-of-operations.  The  tactical  situation  data  is  delivered  by  a  C4ISR- 
system.  This  setup  is  illustrated  in  figure  4. 


Figure  4:  Interactive,  3D  Stereoscopic  Visualization  of  C4ISR-Data. 


The  presentation  is  interactive,  so  that  the  point-of-view  can  be  adjusted  and  modified.  Interesting  subsets 
of  the  scenario  can  be  enlarged  in  order  to  present  details.  Alternatively,  a  more  global  overview  of  the 
situation  can  be  displayed.  Further  functionality  includes  hiding  or  adding  information  by  sending  the 
according  requests  to  the  C4ISR-database.  Interaction  was  implemented  in  different  ways.  A  standard  PC- 
mouse  allows  a  desktop-like  interaction  for  navigation  or  activating  of  drop-down  command  menus. 

Because  it  was  observed  that  an  audience  often  had  difficulties  following  these  actions,  speech  recognition 
was  integrated  into  the  system.  In  this  case,  the  presenter  controls  the  systems  by  a  set  of  command  words. 
This  addition  increased  the  understandability  of  the  actions.  The  setup  is  based  on  available  technology,  so 
that  it  is  possible  to  integrate  it  into  existing  training  concepts  with  little  efforts. 

2.3  Electronic  Sandbox 

For  more  interactive  applications  like  mission  planning  new  media  are  necessary.  In  case  of  the  Electronic 
Sandbox  (ElSa)  at  FGAN/FKIE  a  military  sandbox  as  used  for  training  military  decision  making  served  as 
interaction  metaphor.  ElSa  was  developed  as  an  advanced  display  for  tactical  information  in  mission 
planning,  control  and  rehearsal  [11].  The  setup,  as  shown  in  figure  5,  is  very  similar  to  the  educational 
sandbox.  In  this  case,  a  real  model  is  used  to  present  special  scenarios  in  a  more  realistic  way.  It  gives  a 
birds-eye  view  of  the  situation  of  their  own  and  opponents  units  and  is  very  useful  for  illustrating 
strategies  and  tactics.  We  used  a  semi-immersive  horizontal  projection  table  to  implement  this  metaphor 
[12].  An  image  is  projected  onto  a  horizontal  projection  plane.  Like  with  a  real  sandbox,  team  members 
can  sit  or  stand  around  the  display  and  work  with  the  synthetic  terrain  as  if  it  was  a  real  model. 
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Figure  5:  Sandbox  Education  (left)  and  Electronic  Sandbox  (ElSa)  (right). 


2.4  Augmented  Reality  Sandbox  (ARSa) 

The  ElSa  setup  was  primarily  designed  for  a  stationary  use  in  operation  centers.  The  support  of 
commanders  of  operational  units  and  platoons  requires  a  more  mobile  and  portable  technology.  In  a 
subsequent  research  project  a  prototypic  system  for  applications  in-the-field  was  developed.  It  is  based  on 
the  same  software  framework  as  the  previous  two  systems,  but  applies  technology  from  the  field  of  AR. 
The  system  includes  an  individual  head-mounted  display  (HMD),  notebook  and  special  interaction 
devices.  A  camera  mounted  on  the  HMD  captures  the  real  world  video-images.  The  digitized  images  are 
searched  for  special  patterns  called  markers.  If  a  marker  is  detected,  the  software  calculates  its  relative 
position  and  orientation  to  the  user.  A  virtual  object  is  then  correctly  placed  onto  the  marker  and 
embedded  into  the  video  stream.  The  combined  scene  is  visualized  on  the  individual  HMD.  Figure  6 
illustrates  the  general  idea  of  the  system. 


Figure  6:  Mission  Briefing  in  the  Field  (left)  and  with  the  ARSa  (right). 


Interacting  within  AR  applications  is  more  difficult  than  in  VEs  because  this  technology  extends  the  real 
world  with  virtual  objects.  Hence,  a  developer  of  an  interaction  metaphor  has  to  find  a  way  to  manipulate 
virtual  objects  with  real  interaction  devices.  Many  approaches  have  been  made  over  the  last  years  and  the 
current  direction  is  clearly  direct  interaction  with  the  virtual  parts  of  the  scene.  But  direct  interaction 
brings  along  certain  disadvantages  like  extra  tracking  of  the  input  device,  user’s  mobility  constraints 
caused  by  wired  input  devices  or  occlusion  of  the  markers.  Once  a  marker  is  occluded  and  no  other 
compensation  for  this  state  is  implemented  (i.e.  special  pattern  recognition  for  partially  occluded  markers) 
the  virtual  objects  disappear  from  the  scene  and  interaction  is  impossible. 
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With  respect  to  the  facts  mentioned  above,  a  method  for  an  indirect  input  device  was  implemented.  By 
mapping  a  portable  2D  input  device  correctly  into  a  virtual  3D  scene  a  consistent  interaction  with  virtual 
objects  is  achieved  because  the  interaction  tool  will  never  occlude  a  marker  when  a  user  is  trying  to 
manipulate  the  virtual  object.  The  cursor  that  is  projected  into  the  3D  scene  is  displayed  as  a  virtual  3D 
pointer.  Furthermore,  using  a  common  2D  input  device  does  not  require  extra  training  for  a  potential  user 
because  nowadays  almost  everyone  is  used  to  work  with  a  standard  PC  mouse.  Another  benefit  of  this 
approach  is  that  it  does  not  require  any  calibration  of  input  devices  or  manipulation  of  camera  parameters. 
The  software  framework  which  is  used  for  the  AR  application  as  well  provides  the  opportunity  to  integrate 
more  than  one  AR  client  into  the  existing  application.  The  distributed  architecture  allows  multiple  users  to 
work  in  a  collaborative  AR  application. 

This  collaborative  setup  can  also  be  equipped  with  the  proposed  interaction  metaphor.  Each  user  can 
handle  its  own  input  device  without  distracting  the  fellow  users.  In  order  to  improve  the  cooperation  it  is 
desired  that  each  user  is  always  aware  of  the  position  of  the  corresponding  virtual  pointer.  This  is 
guaranteed  by  initializing  the  different  pointers  with  different  colors,  sizes  or  shapes.  Hence,  each  user  is 
always  able  to  determine  the  position  of  virtual  pointers  which  improves  the  performance  in  a 
collaborative  task. 

Collaboration  in  this  distributed  application  is  supposed  to  be  very  intuitive  because  it  is  always  possible 
to  see  the  team  members,  to  talk  to  them  as  well  as  to  see  their  interaction  with  the  virtual  objects  at  the 
same  time  -  without  any  time  delays.  As  a  matter  of  fact  it  is  also  possible  to  integrate  voice  control  into 
this  setup. 

2.5  VE-Training 

For  a  successful  mission  execution  a  high  level  of  training  is  required.  Today’s  education  and  training 
concepts  include  lectures  and  literature  for  transfer  of  theoretical  knowledge  and  exercises  in  designated 
training  areas  for  the  transfer  of  practical  knowledge.  Innovative  media  like  Virtual  Environments  are 
considered  to  combine  the  benefits  of  both.  They  are  flexible  for  presenting  a  broad  range  of  scenarios, 
and  they  facilitate  a  realistic  presentation  with  natural  interaction.  Moreover,  they  offer  functionality  for 
the  integration  of  actual  C4ISR  information  data  so  that  it  is  available  for  operational  mission  preparation 
as  well. 


Figure  6:  Live-Simulation  in  Bonndorf  (Hammelburg)  and  in  VE. 


In  this  case  the  training  itself  can  rely  on  actual  mission  data  as  training  scenario.  This  is  very  promising 
for  training  efficiency  and  training  transfer.  However,  the  media  and  display  systems  range  from  simple 
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COTS-PC-Systems  to  high-tech  immersive  VE-systems.  In  an  actual  project  described  briefly  in  section 
3.2  we  analyzed  characteristics  and  effects  of  the  applicability  of  different  displays  with  varying  degree  of 
immersion  on  transfer  of  navigational  knowledge. 


3  ERGONOMICS  AND  HUMAN  FACTORS 

Relevant  human  factors  for  designing  a  VE-display  system  issues  are  manifold.  They  span  a  range  from 
open  microergonomic  questions  in  the  areas  of  visualization  and  interaction  to  open  macroergonomics 
issues,  including  multi-factor  phenomenas  like  information  processing  workload,  situational  awareness, 
training  transfer,  and  simulator  sickness. 

3.1  Microergonomic  questions 

The  term  microergonomics  usually  refers  to  product  ergonomics  on  a  basic  level.  In  case  of  a  VE-system, 
this  basic  level  subsumes  analyses  of  visibility  and  interactivity.  Both  topics  have  to  be  analyzed  for  each 
special  display  system  in  order  to  obtain  relevant  reference  data.  In  our  experiments  we  have  studied 
special  areas  of  visualization  and  interaction  with  a  special  focus  on  the  Electronic  Sandbox  and  the 
Augmented  Reality  Sandbox. 

3.1.1  Visualization  and  Interaction  at  the  Electronic  Sandbox  (ElSa) 

In  real  environments,  depth  cues  like  perspective,  size  difference,  occlusion,  shadow,  accomodation, 
convergence  and  disparity  facilitate  an  exact  depth  perception  in  near,  medium,  and  far  range.  Depth 
perception  in  VE  differs  from  reality  [13].  Main  differences  have  their  origin  in  characteristics  of  the 
tracking  and  rendering  system  (e.g.  latency,  inaccuracies),  the  display  itself  (e.g.  resolution,  update  rates), 
or  even  in  visual  deficits  of  the  user  (e.g.  amblyopia,  contradicting  depth  cues).  They  may  lead  to  limited 
or  impossible  depth  perception  in  VEs  and  introduce  additional  ocular  stress. 

A  first  experiment  was  performed  to  analyze  the  effect  of  rendering  on  visual  depth  perception.  As  not  all 
aspects  of  rendering  could  be  included,  four  types  of  rendering,  respectively  display  types  were  specified 
in  advance  of  the  experiment.  The  intended  application  of  terrain  visualization  led  to  the  consideration  of 
wireframe  and  map  texture.  Head-tracking  (motion  parallax)  and  stereoscopy  (binocular  sight)  were  added 
as  further  display  options. 

Significant  effects  of  stereoscopy  (F=13.6,  p<0.01)  and  map  texture  (F=l  1.4,  p<0.01)  on  stereoacuity  were 
found.  Stereoscopy  improves  and  map  texture  reduces  depth  perception.  With  regard  to  reaction  time,  a 
second  4-way  repeated  measures  ANOVA  revealed  significant  effects  of  texturing  (F=7.8,  p<0.01),  head¬ 
tracking  (F=5.7,  p=0.03)  and  stereoscopic  rendering  (F=5.1,  p=0.03).  For  wireframe  no  significant  effect 
is  detected  (F=0.16,  p=0.69).  These  findings  prove  the  benefits  of  stereoscopic  data  presentation  in 
contrast  to  an  accumulation  of  2D-displays  with  elevation  information  (e.g.  by  color  coding).  This  is 
especially  important  for  presenting  complex  additional  textural  or  color-coded  information.  In  this  case 
stereoscopy  brings  benefits  for  depth  perception  and  helps  to  prevent  perceptual  overload. 

Due  to  stimulus  thresholds  for  depth  perception  different  limits  for  stereoscopic  depth  perception  exist. 
The  goal  of  a  second  experiment  was  to  determine  these  limits  of  a  stereoscopic  workbench  in  a  display- 
fixed  coordinate  system.  The  experimental  procedure  for  determining  the  display  limits  was  a  modified 
method  of  constant  stimuli,  which  is  frequently  used  for  stimulus  threshold  analysis.  The  experimental 
virtual  scene  setup  consisted  of  five  objects  in  the  same  fixation  plane.  One  of  these  objects  was  prominent 
to  the  others.  As  a  second  factor,  the  distance  between  fixation  plane  and  projection  plane  varied.  Subjects 
were  instructed  to  react  as  soon  as  they  detect  the  prominent  object  and  afterwards  to  enter  the  result.  This 
result  was  taken  as  measure  for  depth  perception  performance  under  the  specific  experimental  condition. 
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For  determining  the  maximum  stereoacuity  a  significant  effect  of  the  distance  between  fixation  plane  and 
projection  plane  on  stereoacuity  was  found  (F=30.6,  p<0.01).  A  subsequent  trend  analysis  specified  a 
linear  relationship  (F=74.6,  p<0.01).  With  regard  to  the  maximum  distance  before  desegregation  of  the 
picture,  a  significant  effect  was  determined  as  well  (F=5.4,  p<0.01).  A  subsequent  trend  analysis  showed  a 
linear  relationship  (F=29.3,  p<0.01).  For  the  fixation  plane  meeting  the  projection  plane  a  maximum  limit 
of  am=0.43  m  (s=0.06  m)  for  object  elevation  was  found.  At  a  maximum  fixation  plane  elevation  of  0.96 
m  above  projection  plane  no  depth  separation  was  possible.  The  maximum  stereoacuity  deviated  between 
am=0.08  m  (s=0.04  m)  and  am=0.17  m  (s=0.09  m),  dependent  on  the  distance  between  fixation  and 
projection  plane. 

For  activities  at  stereoscopic  projection  table  a  high  ocular  stress  is  expected,  possibly  resulting  in  serious 
negative  after-effects  on  the  visuals  system  of  the  user.  For  this  experiment  subjects  processed  a  visual 
screening  test  before  and  after  the  two  experiments  described  before.  These  experiments  took  about  !4 
hour  and  were  characterized  by  difficult  stereoscopic  viewing  tasks  close  to  depth  perceptional  thresholds. 
Due  to  this,  a  high  visual  workload  resulted.  The  viewing  screening  test  used  in  this  experiment  has  been 
standardized  by  DIN  52880  and  G37  respectively  [14,  15].  The  first  part  comprises  visual  acuity  and  the 
second  part  several  tests  of  binocular  sight.  Based  on  their  values,  ANOVAs  show  no  negative  visual 
after-effects  for  lA  hour  of  activity  at  stereoscopic  projection  tables. 

These  findings  support  the  usability  of  stereoscopic  projection  tables  for  the  visualization  of  geographic 
information,  as  long  as  special  display  bounds  are  considered. 

Implementation  of  interaction  into  a  VE  system  is  possible  in  various  ways.  For  this  application,  manual 
interaction  was  investigated.  It  can  be  implemented  either  by  direct  or  indirect  control,  thereby  using 
different  interaction  devices.  For  the  analysis  of  the  type  of  control  (direct,  indirect),  interaction  device  (pc 
mouse,  space-mouse,  flying  mouse,  pen,  grasp  handle),  and  type  of  handling/grasping  experiments  were 
conducted. 

Looking  at  performance  measures  and  ratings  for  the  3D  interaction  devices,  there  was  a  clear  distinction 
of  interaction  devices  based  on  control  type.  In  each  of  the  three  movement  phases  (target  selection, 
command  selection,  backward  movement)  devices  with  direct  control  (direct  DOF-cursor,  direct  DOF-ray, 
stylus  A,  stylus  B,  grip  handle)  facilitated  better  performance,  that  means  shorter  movement  times,  than 
those  with  indirect  control  (indirect  DOF-cursor,  indirect  DOF-ray,  velocity  cursor,  SpaceMouse™).  In 
contrast  to  this  there  were  no  significant  differences  between  interaction  devices  within  the  same  control 
type  group.  The  same  was  true  for  the  subjective  rating.  As  a  result,  for  the  same  control  type  it  did  not 
matter,  which  interaction  device  to  use  or  what  kind  of  grip  is  applied;  fine  tuning  or  design  of  the  specific 
interaction  device  had  no  significant  effect  on  performance.  On  the  other  hand,  for  the  same  interaction 
device  it  was  important  to  apply  direct  control  rather  than  indirect  control.  This  improves  performance 
significantly.  A  reason  for  this  result  was  that  direct  control  was  more  consistent  than  indirect  control.  In 
case  of  a  stylus,  the  virtual  ’’laser  beam”  extended  a  real  stylus  and  no  visual  or  logical  breaks  are 
perceived.  Consequently,  no  additional  cognitive  resources  were  required  to  map  motoric  output  and 
system  response.  The  missing  differences  between  interaction  devices  of  the  same  control  type  could  be 
based  upon  the  relatively  small  differences  in  handling  or  grasping,  which  are  too  small  for  operators  to 
recognize.  However,  they  could  be  of  interest  for  long-term  studies  looking  at  periods  of  e.g.  a  whole 
working  month. 

It  should  always  be  carefully  checked  if  the  interaction  within  the  application  cannot  be  implemented 
using  a  simple  2D  input  device  as  a  PC  mouse.  This  would  mean  no  performance  or  handling  losses  in 
comparison  with  3D  interactions  devices.  For  applications  requiring  3D  input,  direct  control  with 
consistency  between  interaction  device  and  virtual  counterpart  should  be  implemented.  Indirect  control 
should  be  avoided  when  possible.  According  to  the  results  of  this  study,  special  handling,  grasping,  or 
design  of  interaction  device  brings  no  additional  benefits  for  performance  or  subjective  handling. 
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3.1.2  Interaction  at  the  Augmented  Reality  Electronic  Sandbox  (ARSa) 

Providing  interaction  capabilities  is  crucial  for  state  of  the  art  AR-applications.  Many  different  ways  of 
interacting  have  been  defined  and  implemented  over  the  last  years.  However,  the  influence  of  certain 
technical  characteristics  of  a  typical  AR  system  has  hardly  been  investigated.  Such  system  properties  can 
cause  tracking  errors  and  latency.  They  strongly  affect  the  interaction  as  well  as  the  overall  usability  of 
such  an  application. 

A  common  real-time  AR  system  provides  approximately  25  frames  per  second  (fps)  so  that  the  user  will 
hardly  notice  single  frames  in  the  video  stream.  As  this  is  the  regular  case,  25  fps  is  one  value  for  the 
frame  rate  that  is  analyzed.  A  slight  lower  frame  rate  of  18  fps  was  chosen  to  analyze  the  impact  of  a 
situation  when  a  complex  virtual  object  is  rendered  and  the  application  needs  extra  computational  time. 
The  last  fps-value  to  be  studied  was  set  to  8  in  order  to  test  the  effect  of  a  very  slow  AR  system  on  the 
users.  For  the  intensity  of  a  jittering  object  also  three  values  were  chosen  for  the  first  experiment.  The  first 
one  should  be  clearly  no  extra  jitter  added  to  the  system.  A  slight  boost  of  the  jitter  (medium  value)  was  to 
be  analyzed  in  a  second  test  and  the  last  parameter  was  also  set  to  a  value  that  resulted  in  a  relatively 
strong  jitter  of  the  virtual  map.  Preliminary  tests  showed  that  the  third  and  highest  jitter  value  made  the 
second  experiment  almost  impossible  to  complete  so  only  the  low  and  medium  jitter  was  used. 

Two  different  experiments  were  designed  in  order  to  specify  the  effect.  The  first  one  consists  of  a  simple 
selection  task.  Three  position  landmarks  (spheres)  of  different  sizes  and  colors  appeared  on  the  map  and 
the  user  was  instructed  to  target  the  sphere  with  a  certain  color.  Afterwards,  the  spheres  were  relocated  on 
the  map.  As  soon  as  the  user  selected  the  correctly  colored  sphere  again,  a  selection  menu  was  opened  and 
he  had  to  select  the  correct  color.  The  second  experiment  was  basically  about  the  transfer  of  real 
information  into  the  virtual  object.  The  user  had  to  look  at  a  real  map  (captured  by  the  camera).  The  task 
was  to  transfer  the  information  from  the  real  map  correctly  into  the  virtual  map  as  fast  and  correct  as 
possible. 

The  results  show  that  even  low  frame  rate  does  not  have  a  strong  impact  on  the  performance.  It  is  slightly 
worse  than  a  real  time  system.  A  significant  influence  on  the  accuracy  and  time  does  only  have  the  jitter. 
The  medium  and  the  low  jitter  value  achieve  almost  the  same  performance.  But  the  highest  jitter  strongly 
increases  the  Index  of  Difficulty.  In  the  second  experiment,  the  different  values  of  the  frame  rate  do  not 
have  a  significant  impact  on  the  users'  performance  -  in  this  case  the  number  of  failures  during  the 
experiment.  Another  interesting  result  was  found  after  the  analysis  of  the  Simulator  Sickness  questionnaire 
SSQ  [16].  Users  did  not  complain  about  cybersickness  at  all.  Only  the  concentration  after  an  experiment 
was  worse  than  before  the  experiment.  Some  users  stated  a  slight  exhaustion  after  the  tests  but  there  was 
no  occurrence  of  nausea  or  dizziness. 

3.2  Macroergonomic  questions 

In  contrast  to  microergonomics,  the  term  macroergonomics  refers  to  complex  systems  with  multiple 
relevant  influencing  factors.  In  a  VE  it  would  also  relate  to  multi-factor  effects.  The  most  important  ones 
are  workload,  presence,  situational  awareness,  total  human  C4ISR  system  performance,  and  simulator 
sickness. 

In  our  actual  work  we  examine  these  effects  for  a  VE-training  system  with  different  display  technologies. 
Clear  and  brief,  we  wanted  to  see  if  the  use  of  high-tech  immersive  VE  allows  better  transfer  of 
navigational  knowledge  than  a  low-tech  CRT-display.  Another  aspect  is  the  appearance  of  simulator 
sickness.  It  was  postulated,  that  more  immersive  displays  can  cause  a  larger  amount  of  nausea  and 
disorientation  than  less  immersive  displays. 

In  the  actual  setting  we  used  an  immersive  head-mounted  display,  a  stereoscopic  large-scale  projection 
wall,  which  still  allowed  the  perception  of  peripheral  sight,  and  a  standard  21”  CRT  with  shutterglasses. 
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Subjects  were  instructed  to  localize  landmarks  in  a  synthetic  town,  memorize  landmarks  and  their 
position,  and  report  them  after  12  minutes  exposition  with  free  navigation. 

Analysis  is  still  ongoing,  so  no  final  results  can  be  presented.  However,  it  was  found  that  simulator 
sickness  effects  occurred  only  for  a  very  small  number  of  participants.  Interestingly,  it  seems  to  occur  with 
the  same  likelihood  for  each  display.  With  regard  to  training  transfer  no  final  results  can  be  presented  at 
this  point  of  time. 


4  OUTLOOK 

As  seen  from  our  results,  our  past  research  focus  was  on  microergonomic  topics  and  is  moving  towards 
macroergonomic  topics.  We  need  new  metrics  for  assessing  performance  metrics  to  actually  measure  the 
benefit  of  the  application  of  VE  for  display  of  C4ISR  information. 

But  these  studies  only  serve  as  a  basis  for  determining  more  general  advantages  and  disadvantages  of  AR 
and  VE  technologies  as  a  tactical  situation  display.  By  relying  on  these  findings,  the  displays  can  be 
optimized  and  applied  as  displays  for  a  more  comprehensive  analysis  of  their  effects  on  human  decision 
making  effectiveness.  In  this  case,  differences  in  situational  awareness  and  human  decision  making 
performance  using  traditional  media  on  the  one  hand  and  new,  AR  and  VE  media  on  the  other  hand,  will 
be  investigated.  For  instance,  it  has  to  be  analyzed,  if  the  goal  of  higher  realism  has  a  positive  or  negative 
effect  on  training  effectiveness.  One  outcome  of  this  analysis  will  be  a  more  detailed  recommendation  of 
special  AR  or  VE  technology  for  special  education  and  training  sections. 

But  for  making  full  use  of  the  capabilities  of  the  new  technology,  the  displays  have  to  be  integrated  into 
new  C4ISR  concepts  like  NCW  or  CROP  have  to  be  developed.  They  can  acquire  benefits  from  the  high 
level  of  interaction,  realism  and  intuitiveness  of  the  presentation  of  the  training  scene.  By  incorporating, 
for  instance,  a  continuous  shift  between  a  strategic  overview  and  an  individual,  operational  point-of-view, 
complex  interrelationships  and  situational  development  can  be  displayed  easier  than  with  today’s  media. 
This  development,  however,  has  to  incorporate  today’s  findings  about  handling  and  applicability  of  AR 
and  VE  technology. 
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C4ISR 
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C4ISR  stands  for 

-  Command 

-  Control 

-  Communications 

-  Computer 

-  Intelligence 

-  Surveillance 

-  Reconnaissance 


Central  role  of  C4ISR  systems  is  the  provision  of  information. 
Information  flow  and  exchange  of  information  are  essential. 
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NCW  -  Network-Centric  Warfare 
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Military  doctrine  which  exploits  technological  advances  in 

-  information  technology  and 

-  telecommunication 

to  improve  quality  and  speed  of  military  decision  making. 


Vision: 


-  Share  available  information  throughout  the  forces. 

-  Generate  a  coherent,  accurate  picture  of  battlefield 
(common  operational  picture) 

-  Increase  situational  awareness  in  tactical 
decision  making 

-  "Information  dominance  is  essential  for 
battlefield  dominance" 


'Network 

Centric 

Warfare 

Developing  and  Leveraging 
Information  Superiority 

2^  Edltkxi  (kevKed)  =- 

David  S.  Alberts 
John  J.  Garstka 
t  J  Frederick  P.  Stein 
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The  "human  factor"  in  NCW 
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Massive  amount  of  highly 
dynamic,  multi-source 
information  available 


Information  has  to  be 
perceived  by  and 
catch  the  attention  of  a 
human  commander 
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Applicability  of  special  VE-displays 
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Immersion  &  Interactivity 
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VE-Tactical  Situation  Presentation 
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Short-term  modernization  of 
tactical  situation  presentation  by 
means  of  COTS-components 

VE-technology: 

•  Projection  wall  (Powerwall) 
Charakteristics: 

•  Lecture  metaphor 

•  Portable  display  technology 

•  Dark  environment  required 

•  Stereoscopic  presentation 

•  Interaction  by  inertial  mouse 
and  speech  recognition 
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Electronic  Sandbox  (ElSa) 
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Advanced  TSD  for  staff 
cooperation  during  mission 
planning  and  mission  control 


VE-technology: 

•  Projection  table  (Workbench) 
Characteristics: 

•  Sandbox  metaphor 

•  Stationary  display 

•  Dark  environment 

•  Stereoscopic,  orthogonal 
projection  (e.g.  MOUT) 

•  Dynamic,  natural  interaction 
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Augmented  Reality  Sandbox  (ArSa) 
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Operational  requirements  lead 
to  need  for  a  mobile  display 
technology  for  mission 
planning. 

Technology: 

•  (video)  Augmented  Reality 
Charakteristics: 

•  Sandbox  metaphor 

•  Mobile  technology 

•  Lots  of  technological 
shortcomings  (jitter,  framerate, 
limited  fov,  etc.) 

•  Goal:  Facilitate  a  direct 
interaction 
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VE-Training 
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Merging  Live-Simulation 
(realism)  and  Virtual  Simulation 
(control)  for  better  training  and 
mission  prep. 

Technology: 

•  Virtual  Environment  tech. 
Charakteristics: 

•  Dismounted  simulation 

•  Portable  technology 

•  Immersion  and  Presence  lead 
to  higher  training 
effectiveness  and  mission 
performance 

•  Research  topic  of  this  RTG 
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Results  and  findings  from  case  studies 
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Microergonomics 

-  refers  to  product  ergonomics  on  a  basic  level 

-  Visualization  (e.g.  stereoacuity,  visual  after-effects, ...) 

-  Interaction  (e.g.  modality,  control  devices,  ...) 

Macroergonomics 

-  refers  to  complex  systems  with  multiple  influencing  factors 

-  Workload  (perceptual,  cognitive,  emotional, ...) 

-  Situational  awareness  (measures  for  SA, ...) 

-  System  performance  (task,  measure  of  effectiveness, ...) 

-  Simulator  sickness  (nausea,  sensomotorics,  desorientation) 
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Visualization:  An  overview  of  recent  findings 
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Rendering  /  depth  perception 

Stereoscopie  (sig.+),  map  (sig.-), 

Stereoacuity 

amax  (real:  3‘):  [29,7';  42,6'] 

aopt  (real  5-15“):  [0,63';  1,18'] 

After-effects 

Sehscharfe,  Phorie, 

Fusion,  Stereosehen  (n.sig.) 

Age  effects 

none 

AR-dispiay  shortcomings 

Jitter  (sig.  -),  Framerate  (n.sig.) 
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Interaction:  Another  overview ... 


12 


Target  selection  task  (Fitts') 

Interaction  type 
Direct  vs.  indirect  (sig+) 
Interaction  device 
Pointer  and  PC-Mouse  (sig  +) 
Metaphor  /  Display  technology 

no  sig.  differences 
between  ElSa  and  AREIsa 
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Transfer  of  waypoint  navigation  knowledge 
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Experiment  on  the  effect  of 
different  immersive  display  tech . 

-  Immersive  HMD 

-  Large  FOV  projection  wall 

-  Desktop-CRT 

on 

-  navigation  knowledge  transfer 

-  simulator  sickness 

-  workload 

Analysis  still  ongoing 
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Outlook 
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Microergonomic  topics  are  addressed  by  civil  HF-community 

-  Need  for  transfer  and  adaption  of  findings  to  military  environment 


Macroergonomic  topics,  esp.  in  C4ISR,  require  special  solutions: 

-  High  information  and  emotional  workload  with  (fatal)  consequences  of 
wrong  decisions 

-  AR/VE  offer  high  level  of  interaction,  realism  and  intuitiveness,  but ... 

-  Key  factor:  Definition  and  development  of  measures  of  performance 
and  effectiveness 
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